Aspects of this disclosure are generally related to data storage systems which can be used to maintain relatively large data sets and support a relatively large number of concurrent users. The basic building blocks of a data storage system may include computing and storage nodes such as storage engines, storage arrays or storage servers. The nodes provide access to data storage devices such as disk drives and flash drives. A plurality of computing and storage nodes can be interconnected in order to function together as a cluster. A plurality of clusters can be interconnected by communication links to enable the clusters to function as a single distributed storage system.